Early detection of autism using digital behavioral phenotyping.

Early detection of autism, a neurodevelopmental condition associated with challenges in social communication, ensures timely access to intervention. Autism screening questionnaires have been shown to have lower accuracy when used in real-world settings, such as primary care, as compared to research studies, particularly for children of color and girls. Here we report findings from a multiclinic, prospective study assessing the accuracy of an autism screening digital application (app) administered during a pediatric well-child visit to 475 (17-36 months old) children (269 boys and 206 girls), of which 49 were diagnosed with autism and 98 were diagnosed with developmental delay without autism. The app displayed stimuli that elicited behavioral signs of autism, quantified using computer vision and machine learning. An algorithm combining multiple digital phenotypes showed high diagnostic accuracy with the area under the receiver operating characteristic curve = 0.90, sensitivity = 87.8%, specificity = 80.8%, negative predictive value = 97.8% and positive predictive value = 40.6%. The algorithm had similar sensitivity performance across subgroups as defined by sex, race and ethnicity. These results demonstrate the potential for digital phenotyping to provide an objective, scalable approach to autism screening in real-world settings. Moreover, combining results from digital phenotyping and caregiver questionnaires may increase autism screening accuracy and help reduce disparities in access to diagnosis and intervention.

Exploring Complexity of Facial Dynamics in Autism Spectrum Disorder.

Atypical facial expression is one of the early symptoms of autism spectrum disorder (ASD) characterized by reduced regularity and lack of coordination of facial movements. Automatic quantification of these behaviors can offer novel biomarkers for screening, diagnosis, and treatment monitoring of ASD. In this work, 40 toddlers with ASD and 396 typically developing toddlers were shown developmentally-appropriate and engaging movies presented on a smart tablet during a well-child pediatric visit. The movies consisted of social and non-social dynamic scenes designed to evoke certain behavioral and affective responses. The front-facing camera of the tablet was used to capture the toddlers' face. Facial landmarks' dynamics were then automatically computed using computer vision algorithms. Subsequently, the complexity of the landmarks' dynamics was estimated for the eyebrows and mouth regions using multiscale entropy. Compared to typically developing toddlers, toddlers with ASD showed higher complexity (i.e., less predictability) in these landmarks' dynamics. This complexity in facial dynamics contained novel information not captured by traditional facial affect analyses. These results suggest that computer vision analysis of facial landmark movements is a promising approach for detecting and quantifying early behavioral symptoms associated with ASD.

Blink rate and facial orientation reveal distinctive patterns of attentional engagement in autistic toddlers: a digital phenotyping approach.

Differences in social attention are well-documented in autistic individuals, representing one of the earliest signs of autism. Spontaneous blink rate has been used to index attentional engagement, with lower blink rates reflecting increased engagement. We evaluated novel methods using computer vision analysis (CVA) for automatically quantifying patterns of attentional engagement in young autistic children, based on facial orientation and blink rate, which were captured via mobile devices. Participants were 474 children (17-36 months old), 43 of whom were diagnosed with autism. Movies containing social or nonsocial content were presented via an iPad app, and simultaneously, the device's camera recorded the children's behavior while they watched the movies. CVA was used to extract the duration of time the child oriented towards the screen and their blink rate as indices of attentional engagement. Overall, autistic children spent less time facing the screen and had a higher mean blink rate compared to neurotypical children. Neurotypical children faced the screen more often and blinked at a lower rate during the social movies compared to the nonsocial movies. In contrast, autistic children faced the screen less often during social movies than during nonsocial movies and showed no differential blink rate to social versus nonsocial movies.

A tablet-based game for the assessment of visual motor skills in autistic children.

Increasing evidence suggests that early motor impairments are a common feature of autism. Thus, scalable, quantitative methods for measuring motor behavior in young autistic children are needed. This work presents an engaging and scalable assessment of visual-motor abilities based on a bubble-popping game administered on a tablet. Participants are 233 children ranging from 1.5 to 10 years of age (147 neurotypical children and 86 children diagnosed with autism spectrum disorder [autistic], of which 32 are also diagnosed with co-occurring attention-deficit/hyperactivity disorder [autistic+ADHD]). Computer vision analyses are used to extract several game-based touch features, which are compared across autistic, autistic+ADHD, and neurotypical participants. Results show that younger (1.5-3 years) autistic children pop the bubbles at a lower rate, and their ability to touch the bubble's center is less accurate compared to neurotypical children. When they pop a bubble, their finger lingers for a longer period, and they show more variability in their performance. In older children (3-10-years), consistent with previous research, the presence of co-occurring ADHD is associated with greater motor impairment, reflected in lower accuracy and more variable performance. Several motor features are correlated with standardized assessments of fine motor and cognitive abilities, as evaluated by an independent clinical assessment. These results highlight the potential of touch-based games as an efficient and scalable approach for assessing children's visual-motor skills, which can be part of a broader screening tool for identifying early signs associated with autism.

Complexity analysis of head movements in autistic toddlers.

BACKGROUND: Early differences in sensorimotor functioning have been documented in young autistic children and infants who are later diagnosed with autism. Previous research has demonstrated that autistic toddlers exhibit more frequent head movement when viewing dynamic audiovisual stimuli, compared to neurotypical toddlers. To further explore this behavioral characteristic, in this study, computer vision (CV) analysis was used to measure several aspects of head movement dynamics of autistic and neurotypical toddlers while they watched a set of brief movies with social and nonsocial content presented on a tablet. METHODS: Data were collected from 457 toddlers, 17-36 months old, during their well-child visit to four pediatric primary care clinics. Forty-one toddlers were subsequently diagnosed with autism. An application (app) displayed several brief movies on a tablet, and the toddlers watched these movies while sitting on their caregiver's lap. The front-facing camera in the tablet recorded the toddlers' behavioral responses. CV was used to measure the participants' head movement rate, movement acceleration, and complexity using multiscale entropy. RESULTS: Autistic toddlers exhibited significantly higher rate, acceleration, and complexity in their head movements while watching the movies compared to neurotypical toddlers, regardless of the type of movie content (social vs. nonsocial). The combined features of head movement acceleration and complexity reliably distinguished the autistic and neurotypical toddlers. CONCLUSIONS: Autistic toddlers exhibit differences in their head movement dynamics when viewing audiovisual stimuli. Higher complexity of their head movements suggests that their movements were less predictable and less stable compared to neurotypical toddlers. CV offers a scalable means of detecting subtle differences in head movement dynamics, which may be helpful in identifying early behaviors associated with autism and providing insight into the nature of sensorimotor differences associated with autism.

Computational Methods to Measure Patterns of Gaze in Toddlers With Autism Spectrum Disorder.

Importance: Atypical eye gaze is an early-emerging symptom of autism spectrum disorder (ASD) and holds promise for autism screening. Current eye-tracking methods are expensive and require special equipment and calibration. There is a need for scalable, feasible methods for measuring eye gaze. Objective: Using computational methods based on computer vision analysis, we evaluated whether an app deployed on an iPhone or iPad that displayed strategically designed brief movies could elicit and quantify differences in eye-gaze patterns of toddlers with ASD vs typical development. Design, Setting, and Participants: A prospective study in pediatric primary care clinics was conducted from December 2018 to March 2020, comparing toddlers with and without ASD. Caregivers of 1564 toddlers were invited to participate during a well-child visit. A total of 993 toddlers (63%) completed study measures. Enrollment criteria were aged 16 to 38 months, healthy, English- or Spanish-speaking caregiver, and toddler able to sit and view the app. Participants were screened with the Modified Checklist for Autism in Toddlers-Revised With Follow-up during routine care. Children were referred by their pediatrician for diagnostic evaluation based on results of the checklist or if the caregiver or pediatrician was concerned. Forty toddlers subsequently were diagnosed with ASD. Exposures: A mobile app displayed on a smartphone or tablet. Main Outcomes and Measures: Computer vision analysis quantified eye-gaze patterns elicited by the app, which were compared between toddlers with ASD vs typical development. Results: Mean age of the sample was 21.1 months (range, 17.1-36.9 months), and 50.6% were boys, 59.8% White individuals, 16.5% Black individuals, 23.7% other race, and 16.9% Hispanic/Latino individuals. Distinctive eye-gaze patterns were detected in toddlers with ASD, characterized by reduced gaze to social stimuli and to salient social moments during the movies, and previously unknown deficits in coordination of gaze with speech sounds. The area under the receiver operating characteristic curve discriminating ASD vs non-ASD using multiple gaze features was 0.90 (95% CI, 0.82-0.97). Conclusions and Relevance: The app reliably measured both known and new gaze biomarkers that distinguished toddlers with ASD vs typical development. These novel results may have potential for developing scalable autism screening tools, exportable to natural settings, and enabling data sets amenable to machine learning.

A scalable computational approach to assessing response to name in toddlers with autism.

BACKGROUND: This study is part of a larger research program focused on developing objective, scalable tools for digital behavioral phenotyping. We evaluated whether a digital app delivered on a smartphone or tablet using computer vision analysis (CVA) can elicit and accurately measure one of the most common early autism symptoms, namely failure to respond to a name call. METHODS: During a pediatric primary care well-child visit, 910 toddlers, 17-37 months old, were administered an app on an iPhone or iPad consisting of brief movies during which the child's name was called three times by an examiner standing behind them. Thirty-seven toddlers were subsequently diagnosed with autism spectrum disorder (ASD). Name calls and children's behavior were recorded by the camera embedded in the device, and children's head turns were coded by both CVA and a human. RESULTS: CVA coding of response to name was found to be comparable to human coding. Based on CVA, children with ASD responded to their name significantly less frequently than children without ASD. CVA also revealed that children with ASD who did orient to their name exhibited a longer latency before turning their head. Combining information about both the frequency and the delay in response to name improved the ability to distinguish toddlers with and without ASD. CONCLUSIONS: A digital app delivered on an iPhone or iPad in real-world settings using computer vision analysis to quantify behavior can reliably detect a key early autism symptom-failure to respond to name. Moreover, the higher resolution offered by CVA identified a delay in head turn in toddlers with ASD who did respond to their name. Digital phenotyping is a promising methodology for early assessment of ASD symptoms.

Computer Vision Analysis for Quantification of Autism Risk Behaviors.

Observational behavior analysis plays a key role for the discovery and evaluation of risk markers for many neurodevelopmental disorders. Research on autism spectrum disorder (ASD) suggests that behavioral risk markers can be observed at 12 months of age or earlier, with diagnosis possible at 18 months. To date, these studies and evaluations involving observational analysis tend to rely heavily on clinical practitioners and specialists who have undergone intensive training to be able to reliably administer carefully designed behavioural-eliciting tasks, code the resulting behaviors, and interpret such behaviors. These methods are therefore extremely expensive, time-intensive, and are not easily scalable for large population or longitudinal observational analysis. We developed a self-contained, closed-loop, mobile application with movie stimuli designed to engage the child's attention and elicit specific behavioral and social responses, which are recorded with a mobile device camera and then analyzed via computer vision algorithms. Here, in addition to presenting this paradigm, we validate the system to measure engagement, name-call responses, and emotional responses of toddlers with and without ASD who were presented with the application. Additionally, we show examples of how the proposed framework can further risk marker research with fine-grained quantification of behaviors. The results suggest these objective and automatic methods can be considered to aid behavioral analysis, and can be suited for objective automatic analysis for future studies.

Digital Behavioral Phenotyping Detects Atypical Pattern of Facial Expression in Toddlers with Autism.

Commonly used screening tools for autism spectrum disorder (ASD) generally rely on subjective caregiver questionnaires. While behavioral observation is more objective, it is also expensive, time-consuming, and requires significant expertise to perform. As such, there remains a critical need to develop feasible, scalable, and reliable tools that can characterize ASD risk behaviors. This study assessed the utility of a tablet-based behavioral assessment for eliciting and detecting one type of risk behavior, namely, patterns of facial expression, in 104 toddlers (ASD N = 22) and evaluated whether such patterns differentiated toddlers with and without ASD. The assessment consisted of the child sitting on his/her caregiver's lap and watching brief movies shown on a smart tablet while the embedded camera recorded the child's facial expressions. Computer vision analysis (CVA) automatically detected and tracked facial landmarks, which were used to estimate head position and facial expressions (Positive, Neutral, All Other). Using CVA, specific points throughout the movies were identified that reliably differentiate between children with and without ASD based on their patterns of facial movement and expressions (area under the curves for individual movies ranging from 0.62 to 0.73). During these instances, children with ASD more frequently displayed Neutral expressions compared to children without ASD, who had more All Other expressions. The frequency of All Other expressions was driven by non-ASD children more often displaying raised eyebrows and an open mouth, characteristic of engagement/interest. Preliminary results suggest computational coding of facial movements and expressions via a tablet-based assessment can detect differences in affective expression, one of the early, core features of ASD. LAY SUMMARY: This study tested the use of a tablet in the behavioral assessment of young children with autism. Children watched a series of developmentally appropriate movies and their facial expressions were recorded using the camera embedded in the tablet. Results suggest that computational assessments of facial expressions may be useful in early detection of symptoms of autism.

Impact of a digital Modified Checklist for Autism in Toddlers-Revised on likelihood and age of autism diagnosis and referral for developmental evaluation.

LAY ABSTRACT: This was a project in primary care for young children (1-2 years old). We tested a parent questionnaire on a tablet. This tablet questionnaire asked questions to see whether the child may have autism. We compared the paper and pencil version of the questionnaire to the tablet questionnaire. We read the medical charts for the children until they were 4 years old to see whether they ended up having autism. We found that doctors were more likely to recommend an autism evaluation when a parent used the tablet questionnaire. We think that the tablet's automatic scoring feature helped the doctors. We also think that the doctors benefited from the advice the tablet gave them.

Author Correction: Atypical postural control can be detected via computer vision analysis in toddlers with autism spectrum disorder.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Computer vision analysis captures atypical attention in toddlers with autism.

To demonstrate the capability of computer vision analysis to detect atypical orienting and attention behaviors in toddlers with autism spectrum disorder. One hundered and four toddlers of 16-31 months old (mean = 22) participated in this study. Twenty-two of the toddlers had autism spectrum disorder and 82 had typical development or developmental delay. Toddlers watched video stimuli on a tablet while the built-in camera recorded their head movement. Computer vision analysis measured participants' attention and orienting in response to name calls. Reliability of the computer vision analysis algorithm was tested against a human rater. Differences in behavior were analyzed between the autism spectrum disorder group and the comparison group. Reliability between computer vision analysis and human coding for orienting to name was excellent (intra-class coefficient 0.84, 95% confidence interval 0.67-0.91). Only 8% of toddlers with autism spectrum disorder oriented to name calling on >1 trial, compared to 63% of toddlers in the comparison group (p = 0.002). Mean latency to orient was significantly longer for toddlers with autism spectrum disorder (2.02 vs 1.06 s, p = 0.04). Sensitivity for autism spectrum disorder of atypical orienting was 96% and specificity was 38%. Older toddlers with autism spectrum disorder showed less attention to the videos overall (p = 0.03). Automated coding offers a reliable, quantitative method for detecting atypical social orienting and reduced sustained attention in toddlers with autism spectrum disorder.

Atypical postural control can be detected via computer vision analysis in toddlers with autism spectrum disorder.

Evidence suggests that differences in motor function are an early feature of autism spectrum disorder (ASD). One aspect of motor ability that develops during childhood is postural control, reflected in the ability to maintain a steady head and body position without excessive sway. Observational studies have documented differences in postural control in older children with ASD. The present study used computer vision analysis to assess midline head postural control, as reflected in the rate of spontaneous head movements during states of active attention, in 104 toddlers between 16-31 months of age (Mean = 22 months), 22 of whom were diagnosed with ASD. Time-series data revealed robust group differences in the rate of head movements while the toddlers watched movies depicting social and nonsocial stimuli. Toddlers with ASD exhibited a significantly higher rate of head movement as compared to non-ASD toddlers, suggesting difficulties in maintaining midline position of the head while engaging attentional systems. The use of digital phenotyping approaches, such as computer vision analysis, to quantify variation in early motor behaviors will allow for more precise, objective, and quantitative characterization of early motor signatures and potentially provide new automated methods for early autism risk identification.

Automatic emotion and attention analysis of young children at home: a ResearchKit autism feasibility study.

Current tools for objectively measuring young children's observed behaviors are expensive, time-consuming, and require extensive training and professional administration. The lack of scalable, reliable, and validated tools impacts access to evidence-based knowledge and limits our capacity to collect population-level data in non-clinical settings. To address this gap, we developed mobile technology to collect videos of young children while they watched movies designed to elicit autism-related behaviors and then used automatic behavioral coding of these videos to quantify children's emotions and behaviors. We present results from our iPhone study Autism & Beyond, built on ResearchKit's open-source platform. The entire study-from an e-Consent process to stimuli presentation and data collection-was conducted within an iPhone-based app available in the Apple Store. Over 1 year, 1756 families with children aged 12-72 months old participated in the study, completing 5618 caregiver-reported surveys and uploading 4441 videos recorded in the child's natural settings. Usable data were collected on 87.6% of the uploaded videos. Automatic coding identified significant differences in emotion and attention by age, sex, and autism risk status. This study demonstrates the acceptability of an app-based tool to caregivers, their willingness to upload videos of their children, the feasibility of caregiver-collected data in the home, and the application of automatic behavioral encoding to quantify emotions and attention variables that are clinically meaningful and may be refined to screen children for autism and developmental disorders outside of clinical settings. This technology has the potential to transform how we screen and monitor children's development.

Use of a Digital Modified Checklist for Autism in Toddlers - Revised with Follow-up to Improve Quality of Screening for Autism.

OBJECTIVES: To assess changes in quality of care for children at risk for autism spectrum disorders (ASD) due to process improvement and implementation of a digital screening form. STUDY DESIGN: The process of screening for ASD was studied in an academic primary care pediatrics clinic before and after implementation of a digital version of the Modified Checklist for Autism in Toddlers - Revised with Follow-up with automated risk assessment. Quality metrics included accuracy of documentation of screening results and appropriate action for positive screens (secondary screening or referral). Participating physicians completed pre- and postintervention surveys to measure changes in attitudes toward feasibility and value of screening for ASD. Evidence of change was evaluated with statistical process control charts and χ2 tests. RESULTS: Accurate documentation in the electronic health record of screening results increased from 54% to 92% (38% increase, 95% CI 14%-64%) and appropriate action for children screening positive increased from 25% to 85% (60% increase, 95% CI 35%-85%). A total of 90% of participating physicians agreed that the transition to a digital screening form improved their clinical assessment of autism risk. CONCLUSIONS: Implementation of a tablet-based digital version of the Modified Checklist for Autism in Toddlers - Revised with Follow-up led to improved quality of care for children at risk for ASD and increased acceptability of screening for ASD. Continued efforts towards improving the process of screening for ASD could facilitate rapid, early diagnosis of ASD and advance the accuracy of studies of the impact of screening.